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Abstract 

We study a two armed-bandit algorithm with penalty. We show the convergence of 
the algorithm and establish the rate of convergence. For some choices of the parame- 
ters, we obtain a central limit theorem in which the limit distribution is characterized 
as the unique stationary distribution of a discontinuous Markov process. 
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Introduction 

In a recent joint work with P. Tarres (see ^21 )) we studied the convergence of the so-called 
two armed bandit algorithm. The purpose of the present paper is to investigate a modified 
version of this algorithm, in which a penalization is introduced. In the terminology of 
learning theory (see jl4l I15j ) , the algorithm studied in J2] was a Linear Reward-Inaction 
[LRI) scheme, whereas the one we want to introduce is a Linear Reward-Penalty {LRP) 
procedure. 

In our previous paper, the algorithm was introduced in a financial context as a pro- 
cedure for the optimal allocation of a fund between two traders who manage it. Imagine 
that the owner of a fund can share his wealth between two traders, say A and and that, 
every day, he can evaluate the results of one of the traders and, subsequently, modify the 
percentage of the fund managed by both traders. Denote by X„ the percentage managed 
by trader A at time n (X„ G [0, 1]). We assume that the owner selects the trader to be 
evaluated at random, in such a way that the probability that A is evaluated at time n is 
Xn, in order to select preferably the trader in charge of the greater part of the fund. In 
the LRI scheme, if the evaluated trader performs well, its share is increased by a fraction 
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7n G (0, 1) of the share of the other trader, and nothing happens if the evaluated trader 
performs badly. Therefore, the dynamics of the sequence {Xn)n>o can be modelled as 
follows: 

Xn+l = Xn + 7„+l (l{(7„+i<x„}nyl„+i(l - Xn) - l{C/„+i>X„}nB„+i-'^n) , ^0 = 2; G [0, 1], 

where (f/n)n>i is an iid sequence of uniform random variables on the interval [0,1], 
(resp. Bn) is the event "trader A (resp. trader B) performs well at time n". We assume 
P(An) = p^, P(-Bn) = for n > 1, with p^,Pg £ (0, 1), and independence between these 
events and the sequence {Un)n>i- The point is that the owner of the fund does not know 
the parameters p^, Pg- 

This recursive learning procedure has been designed in order to assign asymptotically 
the whole fund to the best trader. This means that, if say p^ > Pb^ converges to 1 
with probability 1 provided XqG (0, 1) (if < Pg, the limit is with symmetric results). 
However this "infallibility" property needs some very stringent assumptions on the reward 
parameter 7^ (see 1121). Furthermore, the rate of convergence of the procedure either 
toward its "target" 1 or its "trap" is not ruled by a CLT with rate -^TVi like standard 
stochastic approximation algorithms (see JHI)- It is shown in that this rate is quite 
non-standard, strongly depends on the (unknown) values p^ and p^ and becomes very 
poor as these probabilities get close to each other. 

In order to improve the efficiency of the algorithm, one may imagine to introduce a 
penalty when an evaluated trader has unsatisfactory performances. More precisely, if the 
evaluated trader at time n performs badly, its share is decreased by a penalty factor /9n7n- 
This leads to the following LRP - or "penalized two-armed bandit - procedure 

Xn+l = Xn+ 7n+l (l{C/„+i <X„}nA„+i (1 " Xn) - l{(/„+i>X„}nB„+i-'^n) 

-7n+lPn+l (^nl {{/„+! <X„}nA^+, " (1 " -'^n) >X„}nB^_^ J , ^ G N, 

where the notation A^ is used for the complement of an event A. The precise assumptions 
on the reward rate 7^ and the penalty rate 7nPn will be given in the following sections. 

The paper is organized as follows. In Section Q we discuss the convergence of the 
sequence (X„)„>o. First we show that, if /?„ is a positive constant the sequence converges 
with probability one to a limit x* G (0, 1) satisfying x* > ^ if and only if > Pg, so that, 
although the algorithm manages to distinguish which trader is better, it does not assign 
the whole fund to the best trader. To get rid of this limitation, we consider a sequence 
{Pn)n>i which goes to zero so that the penalty rate becomes negligible with respect to the 
reward rate {'y-nPn = o(7n))- This framework seems new in the learning theory literature. 
Then, we are able to show that the algorithm is infallible i.e., if p. > p„, then lim Xn = 1 

n—>oo 

almost surely, under very light conditions on the reward rate 7n (and pn). From a stochastic 
approximation viewpoint, this modification of the original procedure has the same mean 
function and time scale (hence the same target and trap, see ((S|) but it always keeps the 
algorithm away from the trap without adding noise at these equilibria. In fact, it was 
necessary not to add noise at these points in order to remain inside the domain [0, 1]. 
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The other two sections are devoted to the rate of convergence. In Section [21 we show 
that under some conditions (including hm 7n//0n = 0) the sequence Yn = (1 — Xn)/pn 
converges in probabihty to {1—p^)/tt, where vr = p^—Pg > 0. With additional assumptions, 
we prove that this convergence occurs with probability 1. In Section |31 we show that if the 
ratio jn/Pn goes to a positive limit as n goes to infinity, then (l^)„>i converges in a weak 
sense to a probability distribution u. This distribution is identified as the unique stationary 
distribution of a discontinuous Markov process. This result is obtained by using weak 
functional methods applied to a re-scaling of the algorithm. This approach can be seen as 
an extension of the SDE method used to prove the CLT in a more standard framework of 
stochastic approximation (see dHI)- Furthermore, we show that v is absolutely continuous 
with continuous, possibly non-smooth, piecewise C°° density. An interesting consequence 
of these results for practical applications is that, by choosing p„ and 7„ proportional to 
n"-^/^, one can achieve convergence at the rate l/y^, without any a priori knowledge about 
the values of and Pg. This is in contrast with the case of the LRI procedure, where 
the rate of convergence depends heavily on these parameters (see jllj) and becomes quite 
poor when they get close to each other. 

Notation. Let (a„)n>o and (6n)n>o be two sequences of positive real numbers. The 
symbol an ~ 6n means an = bn + o{bn)- 

1 Convergence of the LRP algorithm 

1.1 Some classical background on stochastic approximation 

We will rely on the ODE lemma recalled below for a stochastic procedure (Zn) taking its 
values in a given compact interval /. 

Theorem 1 (a) Kushner & Clark's ODE Lemma (see [9j): Let g : I such that 
Id + g leaves I stable (^). Then, consider the recursively defined stochastic approximation 
procedure defined on I by 

Zn+l = Zn + -fn+l{g{Zn) + Ai?„+i), n > 0, Zq^ I, 

where (7n)n>i is a sequence of [0, l]-valued real numbers satisfying 7n ^ and '}2n>i In = 
+CO. Set N{t) := min{n : 71 -|- • • • -|- 7n+i > t}. If, for every T > 0, 

max 

N{t)<n<N{t+T) 

Let z* be an attracting zero of g in I and G{z*) its attracting interval. Then, on the event 

{Zn visits infinitely often a compact subset of G{z*)} Zn -^^2*. 
""^then for every 7G [0, 1], Id^^g — ^[Id + 3) + (1 — 1)0, still takes values in the convex set 7 



=A'(t)+l 



P-a.s. as t ^ -|-oo. 



(1) 
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(6) The Hoeffding condition (see yj): If {ARn)n>o is a sequence of L°° -bounded 

E_jL 
e T" < +00 for every ■!? > 0, then 

n>l 

Assumption ^\) is satisfied. 

Remark. The monotonous assumption on the sequence 7 can be relaxed into 7n — > 
and sup < +00 

n,k>l 7n 

1.2 Basic properties of the LRP algorithm 

We first recall the definition of the algorithm. We are interested in the asymptotic behavior 
of the sequence {Xn)n&h where Xq = x, with x £ (0, 1), and 

Xn+l = Xn+ 7n+l <X„}nA„+i (1 " Xn) - l{u„+i>X„}nBn+i^nj 

-In+lPn+l {^n^{Ur^+l<X„}nAl^^ " (1 " ^n)'i-{U„+i>X„}nBf^_^-,) , n G N. 

Throughout the paper, we assume that (7n)n>i is a non-increasing sequence of positive 

oo 

numbers satisfying 7„ < 1, ^ 7„ = +co and 

n=l 

Vi? > 0, 2^ e T" < oo, 

n 

and that {pn)n>i is a sequence of positive numbers satisfying jnPn < 1; (f^n)n>i is a 
sequence of independent random variables which are uniformly distributed on the interval 
[0, 1], the events A^, Bn satisfy 

where < Pg < < 1, and the sequences {Un)n>i and l_B„)ra>i are independent. 

The natural filtration of the sequence {Un, ^An, l_B„)n>i is denoted by {J^n)n>o and we set 

t^=Pa -Pb- 
With this notation, we have, for n > 0, 

Xn+l = Xn+ 7„+i (7r/l(X„) + Pn+iK{Xn)) + 7„+lAM„+i, (2) 

where the functions h and k are defined by 

h{x) = x{l — x), = — (1 — + (1 — Pg)(l — x)^, < X < 1, 

AMn+i = Mn+i — Mn, and the sequence (M„)„>o is the martingale defined by Mq = 
and 

AM„+i = l{,7^^^<x„}nA„+i(l - Xn) - l{Un+i>X„}nB„+iXn - TTh{Xn) 

-pn+1 {Xnl{U„^^<X„^}r\Al^^ " (1 " -'^n) l{t/„+i>X„}nB^+i + f^{Xn)) ■ (3) 

Observe that the increments AAf„+i are bounded. 
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1.3 Constant penalty rate 

In this subsection, we assume 

Vn > 1, pn = /O, 

with < p < 1. We then have 

Xn+l = Xn+ 7n+l {hp{Xn) + AM„+i) , 

where 

hp{x) = TTh{x) + pk{x), < X < 1. 

Note that hp{0) = p{l — Pg) > and hp{l) = —p{l — p^) < 0, and that there exists a 
unique x* G (0, 1) such that hp{x*p) = 0. By a straightforward computation, we have 



Xp = T^ZTx II vr 7^ and p 1 



TT - 2p{l -p^) + ^T,'^+ V(l - pb){1 - pa) 
2vr(l - p) 



if vr = or p = 1. 



In particular, x* = l/2if7r = regardless of the value of p. We also have /ip(l/2) = 
7r(l + /))/4 > 0, so that 

x*p > 1/2 if vr > 0. (4) 

Now, let X be a solution of the ODE dx/dt = hp{x). If x(0) G [0, x*], x is non-decreasing 
and lim x{t) = x*. If x(0) G [x*,l], x is non-increasing and lim x{t) = x* . It follows 

that the interval [0, 1] is a domain of attraction for x*. Consequently, using Kushner and 
Clark's ODE Lemma (see Theorem^, one reaches the following conclusion. 

Proposition 1 Assume that pn = p£ (0,1], then 

a.s. * 

Xn — >Xp as n —> OO. 

The natural interpretation, given the above inequalities on x*, is that this algorithm never 
fails in pointing the best trader thanks to Inequality @, but it never assigns the whole 
fund to this trader as the original LRI procedure did. 

1.4 Convergence when the penalty rate goes to zero 

Proposition 2 Assume lim p„ = 0. The sequence (Xn)ne'N almost surely convergent 

n^oo 

and its limit X^o satisfies X^o G {0, 1} with probability 1. 
Proof: We first write the algorithm in its canonical form 

Xn+l = Xn + '^n+l{T^ KXn) + ^Rn+l) with Aii„ = AM„ /9„K;(X„_i). (5) 

It is straightforward to check that the ODE x = h{x) has two equilibrium points, and 
1, 1 being attractive with (0, 1] as an attracting interval and is unstable. 
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Since the martingale increments AM„ are bounded, it follows from the assumptions 
on the sequence (7n)n>i and the Hoeffding condition (see Theorem ^6)) that 

n 

max I 7fcAMjfc| — as t — > +00 

for every T > 0. On the other hand the function k being bounded on [0, 1] and pn 
converging to 0, we have, for every T > 0, 

n 

max I V 7fepfcK(XA,._i)| < ||A;||ro ii(r+7 ) max pk — >0 as t ^ +cx). 

Finally, the sequence (Ai?„)„>i satisfies Assumption Q. Consequently, either X„ visits 
infinitely often an interval [e, 1] for some e > and Xn converges toward 1, or X„ converges 
toward 0. <^ 

Remark 1 If vr = 0, i.e. = Pg, the algorithm reduces to 

Xn+l = Xn + 7„+ip„+i(l - p^){l- 2Xn) + 7„+lAM„+i. 

The number 1/2 is the unique equilibrium of the ODE x = (1— — 2x), and the interval 
[0, 1] is a domain of attraction. Assuming Pnjn = +00, and that the sequence 

{in/ Pn)n>\ is non-increasing and satisfies 

Vt? > 0, ^ exp +00, 

n=l ^ ^"^ 

it can be proved, using the Kushner-Clark ODE Lemma (Theorem^, that lim Xn = 1/2 

n^oo 

almost surely. As concerns the asymptotics of the algorithm when vr = and in = 9 Pn 
(for which the above condition is not satisfied), we refer to the final remark of the paper. 

From now on, we will assume that > Pb- The next proposition shows that the 
penalized algorithm is infallible under very light assumptions on jn and pn- 



Proposition 3 (Infallibility) Assume lim p„ = 0. If the sequence {in/ Pn)n>i is bounded 

n — >cxD — 

and J2nlnPn = 00, o-nd z/vr > 0, we have lim X„ = 1 almost surely. 



Proof: We have from @, since /i > on the interval [0, 1], 

n n 

Xn>XQ + Y, ljPj<Xj_i) + Ij^Mj, n>l. 
i=i i=i 

Since the jumps AMj are bounded, we have 



2 



< C ^ 7J < C sup(7j/pj) ^jPj, 
L2 i=i J'^N 
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for some positive constant C. Therefore, since J^nlnPn = oo, 

L - lim ^„ = so that hm sup ^„ > a.s.. 

Now, on the set {X^^, = 0}, we have 

n 

^7jPjk(Xj_i) 
hm = Ac(0) > 0. 

i=i 

Hence, it fohows that, stih on the set {X^^, = 0}, 

hmsup — — - — > 0. 

i=i 



Therefore, wc must have W{X^. = 0) = 0. 
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The foUowing Proposition wih give a control on the conditional variance process of 
the martingale {Mn)neN which will be crucial to elucidate the rate of convergence of the 
algorithm. 

Proposition 4 We have, for n>0, 

E(am2+i I <p^(i-x„) + p2^i(l-pj. 

Proof: We have 

AM„+i = Vn+l - E{Vn+l I J'n) + Wn+1 - E{Wn+l \ J'n), 

with 

Vn+l = l{!7„+i<X„}n^„+i(l - ^n) - i{U„+i>X„}nB„+iXn 

and 

Wn+l = -Pn+l {Xn'i-iUn+iKXr.jnA-^^^ " (1 " ^n)l{C/„+i>X„}nB^_^ J • 

Note that Vn+iWn+i = 0, so that 

E (AM2^i I J^n) = ^{V^+l I ^n) + ^{W^+l I -^n) " (E(K+i + Wn+1 \ Tn)? 

< E(v;V| J-„) + E(W^„\i| 

Now, using < and Xn < 1 , 

E(V^„\l|^n) = pM^-Xn?+PBi^-Xn)X^ 

< PA(l-^n) 

and EiW^^,\J^n) = p2_^i(x3(l-pJ + (l-X„)3(l-pj) 

< Pn+l(l-Ps)- 

This proves the Proposition. 
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2 The rate of convergence: pointwise convergence 
2.1 Convergence in probability 



Theorem 2 Assume 



lim p„ = 0, lim — = 0, Vp„7n = oo, p„ - = o(p„7„). (6) 

n — >oo n — *oo ^ — ^ 

Then, the sequence ((1 — Xn) / Pn)n>i converges to (1 — p^)/'/r in probability. 

Note that the assumptions of Theorem |2 are satisfied if 7^ = C/n" and pn = C /n^ , with 
C,C' > 0, < r < a and a + r < 1. In fact, we will see that for this choice of parameters, 
convergence holds with probability one (see Theorem ISJ. 
Before proving Theorem |51 we introduce the notation 

Y -1^ 



Pn 



We have, from 



1 - -'^n+l = 1 - -^n - 7n+l7rX„(l - X„) - /9„4.17„+1k(X„) - 7„+iAM„+l 
1 - Xn 7„+l , . , . 7n+l . ,^ 

= 7rA„(l - A„) - 7„+ik(A„) AM„+i. 

Pn+1 Pn+l Pn+l Pn+l 

Hence 

Yn+i = y„ + (1 - Xn) {-^ - — - ^vrX„') - 7n+l«(X„) - ^AAf„+i 

VPn+1 Pn Pn+l J Pn+l 

Yn+l = (1 + 7n+ien " T^nln+lXn) - ^n+ln{Xn) AM„+i, 

Pn+l 

where 

En = and 7r„, = vr. 

7n+l \Pn+l PnJ Pn+l 

It follows from the assumption p„ — Pn-i = oipn^ln) that lim e„ = and lim 7r„ = vr. 

Lemma 1 Consider two positive numbers vr^ and vr^ wii/i 0<7r^<7r<7r^<l. Given 
I G N, let 

= inf{n > I I 7r„X„ - e„ > 7r+ or 7r„X„ - e„ < vr"}. 

• lim ¥{J = 00) = I, 

• forn> I, ife+ = nfc=z+i(l - 7r+7fc) and 9^ = Uk=l+i{'^ - ^^Ik), 

n/\v^ nAu' 

j^<yi- i: ^-iXk-i) - E T^^^^ (7) 

nAy' k=l+l k=l+l 
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and 

nAu'' nAu' 

^>yi-j: S-(^^-i) - E -^AM,. (8) 

^nAu' k=l+l k=l+l P^'^k 

Moreover, with the notation ||A;||oo = supo<a;<i 1/^(2^)1, 

n>l V ^ ^/ vr 

Remark 2 Note that, as the proof will show, Lemma ^ I'emains valid if the condition 
lim 7n/pn = in © is replaced by the boundedness of the sequence ('~fn/ Pn)n>i- In 

n — *cxD ~ 

particular, the last statement, which implies the tightness of the sequence (Yn )«,>!; will 
be used in Section |31 

Proof: Since lim (7r„X„ — e„) = vr a.s., we clearly have lim P(z^' = oo) = 1. 
On the other hand, for I < n < u'', we have 

Yn+i < Yn{l - 7n+ivr") - 7„+ik(X„) AMn+l 

Pn+1 

and 

Yn+l > Ynil - 7„+l7r"*') - 7„+ik(X„) 

Pn+1 

so that, with the notation 9:^ = Ylk=i+ii^ ~ ^^7fc) and 9^ = Ylk=i+i{^ ~ ^ 7A:)) 

7n+l /-^ N 7n+l A 71,1- 

Pn+l^n+l 

and 

Pn+lfn+1 

By summing up these inequalities, we get Q and (jHJ. 
By taking expectations in ((7|), we get 



Yn+l 


< 


Yn 






9n 


Yn+l 


> 


Yn 






On 



y nAv- 

< ^Yi + \\k\U^ ^ 

"nAul k=l+l ^k 

nAu^ 



1 



< ^Yi + 



vr 



We then have 



nYnl{..=oo})=0-^[^l{..=oo}] < 0-¥^ 
\ nAv^ / nAv^ 

< 0n(m + 



vr 
vr 
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Lemma 2 Let {9n)neN be a sequence of positive numbers such that On = nfc=i(l ~ Vlk) 
for some p G (0, 1). The sequence {On X]fc=i 'sj^p^^^^ ^ converges to in probability. 

Proof: It suffices to show convergence to in probability for the associated conditional 
variances defined by 

n 2 

Tn = elY,J^¥.(^Ml\rk-i). 

k=l ^kPk 

We know from Proposition |3] that 

E(am||^,_i) < p^{i-Xu-i)+pl{l-p,) 
= PApk-iYk-i + pI{^-Pb)- 
Therefore, r„ < p^rf^ + {I - Pg)Ti?\ where 

n 2 
k=l ^kPk 

and 

n 2 

7^(2) ^n2Sr^Jk 
k=l ^k 

We first prove that hm T^^^ = 0. Note that, since p'jk < 1, 

1 1 _ 2p7fc - p'^jI -fk 



n2 a2 a2 — P o2 ■ (^) 

f^fc ^k-1 ^k ^k 

Therefore, 



and lim Ti^^ = follows from Cesaro's lemma. 

n^oo " 

We now deal with Ti^\ First note that the assumption pn — Pn-i = o{pn'yn) implies 
lim pn/pn-i = 1, so that, the sequence (7n)n>i being non-increasing with limit 0, we 

n — >oo ~ 

only need to prove that lim = in probability, where 

n 2 
k=l ^kPk 

Now, with the notation of Lemma Q we have, for n > I > 1 and e > 0, 

(n 2 \ 

1 n 2 

< P(z.'<oo) + -02^-^E(ya{.<=oo}; 

e fc=l ^kPk 
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Using Lemma n lim ^n/ Pn = ^ and ®, we have 

n^oo 

n 2 

We also know that lim P(i^' < oo) = 0. Hence, 

/— >oo 

JimP(r«>e)=0. ^ 
Proof of Theorem [21 First note that if 9n = 11^=1(1 - Plk), with < p < 1, we have 

Hence, using lim Xn = 1 and = —(1 —p^)-, 



n 



hm 



Going back to and Q and using Lemma |2l with p = vr^ and vr , and the fact that 
hm P(i/' = oo) = 1, we have, for ah e > 0, lim > + e) = lim P(y„ < 

1 — T) 

— e) = 0, and since vr'^ and vr^ can be made arbitrarily close to vr, the Theorem is 

proved. 

2.2 Almost sure convergence 

Theorem 3 In addition to we assume that for all (3 £ [0, 1], 

InP^ - In-lPn-l = O^nPn), (10) 

and that, for some tj > 0, we have 
Then, with probability 1, 

l-Xn 1-p^ 



lim 



Pn TT 



Note that the assumptions of Theorem |31 are satisfied if 7„ = Cn'"" and pn = C'n~'^, with 
C,C' > 0, < r < a and a + r < 1. 

The proof of Theorem |21 is based on the following lemma, which will be proved later. 
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Lemma 3 Under the assumptions of Theorem \^ let a £ [0, 1] and let {On)n£>i be a 
sequence of positive numbers such that On = YVk=ii^ ~Vlk), for some p G (0,1). On the 
set {sup„(/9°y„) < oo}, we have 

lim 9nP^~^ V jAMfc = a.s., 

where r] satisfies 

Proof of Theorem El We start from the following form of 0: 

1 - Xn+l = (1 - Xn){l - 7„+i7rX„) - Pn+lln+lK{Xn) - 7„+iAM„+i. 

We know that lim Xn = 1 a.s.. Therefore, given vr^ and tt~ , with < vr^ < vr < -7r+ < 1, 

n— ►oo 

there exists / S N such that, for n > I, 

1 - Xn+l < (1 - Xn){l - 7n+lVr") - Pn+lJn+lK{Xn) - 7n+lAM„+i 

and 

1 - Xn+l > (1 - Xn)il - 7n+lVr+) - Pn+l7n+lK{Xn) - 7n+lAM„+i, 

SO that, with the notation 6^ = nfc=i+i(l — '^'^Jk) and 6~ = nfc=i+i(l — "^"Tfc)) 

1 - Xn+l ^ ^ - Xn Pn+lln+1 , ^ n 7rt+l . , ^ 
< t^\Xn) AM„_li 

^n+\ ^n+l 

and 

^ —E+ Z+ ^(^n) - -^AMn+l. 

^n+1 "n+1 ^n+1 

By summing up these inequalities, we get, for n > I + 1, 



k=l+l k k=l+l k 



and 



Hence 



and 



k=l+l "k k=l+l "k 

Yn < ^(1 - X,) - ^ E - - E ^^M, (12) 

k=l+l ^k k=l+l ^k 

yn>^(l-XO-^ E ^-^<Xk^i)-^-^ E ^AMfc. (13) 

fe=/+l ^fc P'^ k=l+l ^k 

We have, with probability 1, lim k(X„) = k(1) = — (1— p^), and, since X^^i Pn7n = +oo, 

n n 

E ^.(x,_,)--(i-Pj E (14) 
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On the other hand, 



Pkik ^ J_ /I 1 



k=l+l "fc ^ k=l+l \"k "k-1, 



TT 



1 Pn 



(15) 



TT y„ 

where we have used the condition pk— Pk-i = o{pk1k) ■ We deduce from ((TH) and (fT3)) that 

Q- 9+ 
and, also, that lim — = 0. By a similar argument, we get lim — = and 

It follows from LemmaOl that given a£ [0, 1], we have, on the set Ea := {supJ^p'^Yn) <oo}. 



Q — ?7 



k=l 



Together with (|12() and H13|) this implies 
. lim y„ = (1 - p^)/7r a.s, if < 0, 



a — rj 



• lim y„i9n^ = a.s., if ^ > 0. 

We obviously have P(-E'q.) = 1 for a = 1. We deduce from the previous argument that 
if ¥{Ea) = 1 and ^ > 0, then P(^„') = 1, with q' = ^ - 1. Set ao = 1 and 
Ofc+i = "'''2^ — 1. If ao < "^5 we have lim Yn = (1 — p^)/7r a.s. on Ea^. If > l^t j 
be the largest integer such that aj > r] (note that j exists because lim < 0). We have 

k—too 

^{Ea^+i) = 1, and, on E^^^-^, lim Yn = (1 —Pj^)/tt a.s., because ^ ^0- 

We now turn to the proof of Lemma |31 which is based on the following classical martin- 
gale inequality (see remark 1, p. 14 for a proof in the case of i.i.d. random variables: 
the extension to bounded martingale increments is straightforward). 

Lemma 4 (Bernstein's inequality for hounded martingale increments) Let (Zj)i<j<„ he a 
finite sequence of square integrable random variahles, adapted to the filtration {J^i)i<i<n, 
such that 
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1. EiZi \J^i_i) = 0,i = l,...,n, 

2. E{Zf\Ti_i)<ali = l,...,n, 

3. \Zi\ < An, i = l,... ,n, 

where af, . . . , a^, A„ are deterministic positive constants. 
Then, the following inequality holds: 



VA > 0, P 



1=1 



> A < 2exp -- 



withbl = Y.Ucrl 

We will also need the following technical result. 

Lemma 5 Let {6n)n>i be a sequence of positive numbers such that 9n = Yl^=i{l —p^k), 
for some p G (0, 1) and let {£,n)n>i be a sequence of non-negative numbers satisfying 



In^n - 7n-lCn-l = o(7nCn)- 



We have 



n 2 



E 



k=l k 



2pel 



Proof: First observe that the condition 7„^„ — 7n-iCn-i = 0(7^^^) implies 7„^„ 
7n-iCn-i S'lid that, given e > 0, we have, for n large enough, 

7nCn — 7n-lCn-l > — £7n^n 

> -£7n-l7n^n, 

where we have used the fact that the sequence (7„) is non- increasing. Since ^n^n 
7n-iCn-i) we have, for n large enough, say n > no, 

7nCn > 7n-l^n-l(l - 2e7n-l)- 

Therefore, for n > uq, 

n 

In^n > 7noCno H ~ 2£7fe_i). 

k=no+l 

Prom this, we easily deduce that lim "yn^n/Gn = 00 and that I]n7n^n/^n = 
Now, from 

1 1 _ 27fcp - 7|p^ 



01 



we deduce (recall that lim 7„ = 0) 

n— ►oo 

lUk Ikik ( 1 



01 



2p \el ^Li. 
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and, since Y^nllin/Ol = oo, 



vie. ^ . 



k=l "k-X/ 

where, for the first equahty, we have assumed .^o = 0, and, for the last one, we have used 
again 7„^n - 7„_i^„_i = o(7^^„). 

Proof of Lemma El Given /U > 0, let 

v^J, = inf{n > | /9"y„ > /z}. 

Note that {sup„ /9^y„ < oo} = U/,>o{^m = 
On the set {f^ = oo}, we have 

" n 



fc=i fc=i 



We now apply Lemma with Zi = ^l|j<j,^}AMj. We have, using Proposition^ 



where we have used the fact that, on {i < z^„}, pf_^Yi^i < fi. Since lim p„ = and 

n — ^oo 

lim Pn/pn-i = 1 (which follows from pn — Pn-i = oi'jnPn)), we have 

n — >oo 

E(Z2 I < af, 

with af = C^'^^^^ — , for some > 0, depending only on Using Lemma [3 we have 

i 



InPr, 

i=l 

On the other hand, we have, because the jumps AMj are bounded. 



E2 n '"^yn 
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for some C > 0. Note that 



Ik /6k 



Ik 



in ((ini)), we have, for A; large enough, 7^. - 7^.1 > -pjklk-i, so that jk/lk-i > 1 -P7fc> 
and the sequence (jn/dn) is non-increasing for n large enough. Therefore, we have 

sup \Zi\ < An, 
l<i<n 



with An = C^n/Gn for some C>0. Now, applying Lemma|l]with A = Ao/On ^ /6'ni we get 



fc=l 



> AoPn 




Cip: 



,2-a+T] 



1_ 1 



< 2exp -C4 



7n 



where the positive constants Ci, C2, C3 and C4 depend on Aq and p, but not on n. 
Using and the Borel-Cantelli lemma, we conclude that, on {f^ = 00}, we have, for n 
large enough, 



< XoPn ^ , a.s.. 



and, since Aq is arbitrary, this completes the proof of the Lemma. 



3 Weak convergence of the normalized algorithm 

Throughout this section, we assume (in addition to the initial conditions on the sequence 

(7n)neN) 

7n - 7L1 = 0(7^) and — =g + o(7„), (16) 

Pn 

where 5^ is a positive constant. Note that a possible choice is 7^ = agj ^fn and pn = a/ y/n, 
with a > 0. 

Under these conditions, we have /?„ — pn-i = 0(7^), and we can write, as in the 
beginning of Section [Jl 

Yn+i = y„ (1 + 7„+ie„ - 7r„7„+iX„) - 7„+ik(X„) — AM„+i, (17) 

Pn+l 

where lim e„ = and lim vr^ = vr. As observed in Remark |21 we know that, under the 

n— >oo n— >oo 

assumptions (|TH|) . the sequence (yn)ra>i is tight. We will prove that it is convergent in 
distribution. 
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Theorem 4 Under conditions ild]) . the sequence (Yn = (1 — ^n)//5n)nGN converges weakly 
to the unique stationary distribution of the Markov process on [0, +00) with generator L 
defined by 

Lf{y) = p,y ^^^ ^ ~ ^^^^ + (1 - - P^y)f'iy), y>0, (18) 

for f continuously differentiable and compactly supported in [0, +00). 

The method for proving Theorem |1] is based on the classical functional approach to central 
limit theorems for stochastic algorithms (see Bouton |2j, Kushner JOl) Duflo 0). The 
long time behavior of the sequence (y„) will be elucidated through the study of a sequence 
of continuous-time processes y(") = (Y^'^^)t>o, which will be proved to converge weakly 
to the Markov process with generator L. We will show that u has a unique stationary 
distribution, and that this is the weak limit of the sequence (i^)nGN- 
The sequence Y^^'^ is defined as follows. Given n G N, and i > 0, set 

^/"^=i"iV(n,*), (19) 

where 

N{n, t) = min lm>n\ ^ -fk+i >t>, 

[ k=n J 

SO that N{n,0) = n, for t S [0,7„4.i), and, for m > n + 1, N{n,t) = m if and only if 

k=n+l lk<t < l^k=n+l 7fc- 

Theorem 5 Under the assumptions of Theorem^ the sequence of continuous time pro- 
cesses (y'^"'')„eN converges weakly (in the sense of Skorokhod) to a Markov process with 
generator L. 

The proof of Theorem [S] is done in two steps: in section 13.11 we prove tightness, in 
section ISI21 we characterize the limit by a martingale problem. 

3.1 Tightness 

It follows from H17|) that the process admits the following decomposition: 

y /") = y„ + + (20) 



with 



and 



N{n,t) 



k=n+l 



N{n.t) 
k=n+l 
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The process {M^^^)t>o is a square integrable martingale with respect to the filtration 
(J^i"^)t>o, with jFj^"^ = TN{n,t), and we have 

N(n,t) 2 

<M(^^>t= (-) E(AM||^,_i). 

k=n+l ^P'^^ 

We already know (see Remark that the sequence (l^n)neN is tight. Recah that in order 
for the sequence (M^")) to be tight, it is sufficient that the sequence (<M(")>) is C-tight 
(see [7], Theorem 4.13, p. 358, chapter VI). Therefore, the tightness of the sequence (y^")) 
in the sense of Skorokhod will follow from the following result. 

Proposition 5 Under the assumptions Mb]) , the sequences (5(")) and (<mW >) are 
C-tight. 

For the proof of this proposition, we will need the following lemma. 

Lemma 6 Define as in Lemma^ for I G N. There exists a positive constant C such 
that, for all l,n,N € N with I < n < N , we have 



VA > 1, P sup \Yj - y„| > A < F{iy^ <oo) + C- 

\n<j<N J 



(i + Eyz)(Efc=n+i7fc 

A 



Proof: The function k being bounded on [0,1], it follows from ()17|) that there exist 
positive, deterministic constants a and b such that, for all n G N, 



7n+i(a + bYn 



7n+l 
Pn+1 



AMn+1 < Yn+l -Yn< 7n+i(a + hY^ 



7«+i 

Pn+1 



AM 



n+l- 



We also know from Proposition ^ that 

E (aM^^i I J^n) < PaPuYu + (1 - Pb)p 
From (|21l) . we derive, for j > n, 



n+l- 



(21) 



(22) 



\Yj -Yn\< 7fc(a + bYk-i) + 

k=n+l 



k=n+l 



Let Yfc = yfcl|^<;j^i| and AM^ = l{^<^i|AMfc On the set = oo, we have Yfc-i = ^fe-i 
and AMfc = AM^. Hence 

pf sup \Yj-Yn\>\\ < P(i^' < cx)) + P I V 7fe(« + ^n-i) > A/2| + 

V<1<N J \k=n+l J 



P I sup 

n<j<N 



k=n+l 



Pk 



> A/2 
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We have, using Markov's inequality and Lemmas 



N 



N 



k=n+l 



< - a + 6supE(Yfcl|^i^^| 
A V k>l ^ 



/ k=n+l 



A 



< -ibEYi+b^- 



1T 



N 

a) E 7fe- 

k=n+l 



On the other hand, using Doob's inequahty. 



16, 



sup 


E -^Mk 


> A/2j 


\n<j<N 


k=n+l 





N 



A' ,^^1 Pk 

k=n+l 

16. A ll 



^ E Ji{fc<.}(p^Pfe-in-i + (i-Ps)/oi)- 



k=n+l Pk 

Using lim(7„/p„) = g, pk~i ~ Pk, hm/9.„ = and Lemmas we get, for some C > 0, 



P I sup 

n<j<N 



E -^M, 

k=n+l P^ 



> A/2 <C 



(i+Eyz)(E 



k=n+l Ik 



A2 



and, since we have assumed A > 1, the proof of the lemma is completed. (j. 

Proof of Proposition El Given s and with < s < we have, using the boundedness 
of K, 

N{n,t) 

|5(n)_^(n)|< ^ 7,(a + 6n_i) 

fc=Ar(n,s)+l 

for some a, 6 > 0. 

Similarly, using (|^ . we have 

N{n,t) 

KM^'^ht - <m(")>,| < ^kia + b'Yk-i) 

k=N{n,s)+l 

for some a' ,b' > 0. These inequalities express the fact that the processes and <mW> 
are strongly dominated (in the sense of [7j, definition 3.34) by a linear combination of the 
processes X^'^) and where X^^"^ = 7fe and Z,^") = ES'+i 7fcn-i- Therefore, 

we only need to prove that the sequences and are C-tight. This is obvious 

for the sequence X^^\ which in fact converges to the deterministic process t. We now 
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prove that is C-tight. We have, for < s < t < T 

/ X N{n,t} 

< sup ya j: 7. 

\n<j<N(n,T) ) fc=;v(n,.)+l 

< (t-S + 77V(n,s)+l) sup Yj 

n<j<N{n,T) 

< (i-s + 7„+i) sup Yj, 

n<j<N{n,T) 

where we have used J2k^+i 7fe ^ * ^^'^ ^ ^ J2k=^+]^^ and the monotony of the sequence 

(7n)n>l- 

Therefore, for 5 > 0, and n large enough so that ^n+i ^ <5, 

P( sup |zf ^ - > r/) < P( sup Yj> 



0<s<t<T,t-s<5 J \n<j<N{n,T) ^ + % 



n+1 



We have, from Lemma IHl 
P 



+p( sup |F^.-y„|>^]. 

\n<j<N(n,T) 40 J 



( \ ACS 

sup |y^.-y„|>^ < P(i.^<oo) + (l+EYO 7fc 

< p(i.' < oo) + — — (1 + EyO- 

We easily conclude from these estimates that, given T > 0, e > and > 0, we have for 
n large enough and b small enough, 



P( sup \zf -Zf \ >r/) <e, 

\^<B<t<T,t-S<& ) 

which proves the C-tightness of the sequence (Z*^")). ^ 
3.2 Identification of the limit 

Lemma 7 Lti f be a function with compact support in [0,+oo). We have 

E {f{Yn+l) - f{Y^) I Tn) = ln+lLf{Y^) + 7n+l^n, ^ G N, 

where the operator L is defined by 

Lf{y)=p,y^^^^^-^^^^ + {l-p^-p^y)fiy), y > 0, (23) 
and the sequence {Zn)n£fi satisfies lim = in probability. 
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Proof: From (|T7)) . we have 

Yn+l = Yn+ 7„+i(-k(1) - irYn) - ^^^^AM„+i + 7„+lCn 

Pn+1 

= 7n+l(l - Pa- ^^n) " ^^^AM„+i + 7„+lCn 

Pn+1 

= Yn + 7n+i(l - El - vry„) - g^Mn+i + 7n+lCn + f 9 " AM„+i, (24) 

where Cn = /^fl)— '^f-^^nl+^nf'?'"— ('''"n-'^n— £n))i SO that Cn IS ^n-measurable and hm Cn = 

n— >oo 

in probabihty. Going back to ©, we rewrite the martingale increment AM„+i as follows: 

AM„+1 = -Xn (l{[/„+i>X„}nB„+i -Pb(1 - -'^n)) + /5n5^n <X„}nA„+i - Pa^u 



Yn+l =Yn + 7n+l(l " Pa " + Cn) + + AM„+i, 

= gXn U{U„+i>X„}nB„+i -Pg(l " -'^n) 



Hence, 
where 
and 

AM„+i = [g - ) AM„+i - gpnYn (l{[/„+i<x„}nA„+i - JU-'^™) 

\ Pn+1/ ^ ' 

+9Pn+l {Xnl{u„+i<X„}nA^^^^ " (1 " ^n) l{[/„+i>X„}nB^^i + '«(-'^« 

Note that, due to our assumptions on 7„ and pn, we have, for some deterministic positive 
constant C, 



AM, 



n+l 



< C7„+i(l + y„), ugN. (25) 
Now, let 

= + 7n+l(l " ?U " + Cn) and Yn+l = Yn + 

SO that Yn+l = Yn+l + AM„+i. We have 

f{Yn+l) - f{Yn) = f{Yn+l) - /(K+l) + /(K+l) " f{Yn). 

We will first show that 
f{Yn+i) - f{Yn+i) = /'(y„)AM„+i + 7n+i7;+i, where P- lim E(T„+i | ^„) = 0, (26) 

n — >oo 

with the notation P- lim for a limit in probability. Denote by w the modulus of continuity 
of /': 

w{5)= sup \f{y)-f{x)l <5>0. 
\x-y\<5 
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We have, for some (random) G (0, 1), 

= f'{Yn)AMn+l + Vn+l, 

where K+i = (/'(?n+i + OAMn+i) - /'(f;)) AMn+i- We have 

\Vn+i\ < w + |AM„+i|) |AM„+i| 

< Cu;(|e„+i| +C7„,+i(l + y„))7„+i(l + y„), 

where we have used Yn+i = Yn + S,n+i and In order to get (j^U, it suffices to prove 

that Mm E(?i;(|^„+i| + C7„+i(I + !"„)) I J^n) = in probabihty. On the set {Un+i > 

n — ^oo 

Xn}nBn+i, we have |^„+i| = gXn (1 - Pg(l - X„)) < g, and, on the complement, |C„+i| = 
gXnPgil - Xn) < g{l - Xn). Hence 

E(w;(|en+i| + C7„+i(l + y„)) I ^„) < p^{l-Xn)wig + Cjn+iil + Yn)) 

+ {1 - p^il - Xn))w (Yn) , 

where Yn = g(l — Xn) + C7„+i(l + !"„). Observe that hm 1^ = in probabihty (recall 

n — »oo 

that lim Xn = 1 almost surely). Therefore, we have (|26j) . 
We deduce from E(AM„+i \ J^n) = that 

E (/(y„+i) - /(y„) I .^„) = 7n+iE(r„+i \Tn) + E (/(i;+i) - /(i;) | .f„) , 

so that the proof will be completed when we have shown 

P- lim E ( - /(^») - 7n+iL/(i;) I ^\ ^ ^ ^27) 

"^^^ 7n+l / 



We have 



Hence 
with 



E{f{Yn+l)\J^n) = E[f{Yn+Cn+l)\Tn 

= p^{l-Xn)f{Yn + gXn{l-pA^-Xn))) 

+ (1 -p^(l - Xn))f{Yn-gXnP^{l " X„)) 
= PsPnYnf{Yn + 9Xn{l-Ps{l-Xn))) 

+ {1 - p^PnYn)f{Yn - gXnP^il - Xn)). 



E (/(y„+l) - /(y„) I J^n) =Fn + Gn, 
Fn = p^PnYn (/(K + gXn{l - p^il " X„))) - /(y„) 
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and 

For the behavior of Fn as n goes to infinity, we use 

P- lim (Yn + gXnil - (1 - Xn)) - Yr, - g) = 0, 

n— >oo \ / 

and hm pn/jn+i = 1/5, so that 

p _ „ y fiYn+g)-f(Y„) 
P- hm ^ = 0. 

For the behavior of we write, using hm Pn/ln+i = 1/5 again, 

n — 5-00 

Yn - gXnPg (1 - Xn) = Yn + 7n+l (1 " Pa " + Cn) " gPeXuRnYn 

= Yn + 7n+l(l - Pa - Pa^") + ln+ir}n, 

with P- hm„_*oo = 0, so that, using the fact that / is C"^ with compact support and the 
tightness of (Yn)-, 

p_ ^.^ Gn-{l-p,-p,Y^)nYn) ^ 

7n+l 

which completes the proof of 1)27^ . <). 

Proof of Theorem EJ As mentioned before, it follows from Proposition El that the 
sequence of processes (y^")) is tight in the Skorokhod sense. 

On the other hand, it follows from Lemma [7| that, if / is a function with compact 
support in [0, +00), we have 

n n 
f{Yn) = f{Yo) + lkLf{Yk-l) + ^kZk-l + Mn, 
k=l k=l 

where (M„) is a martingale and (Zn) is an adapted sequence satisfying P- lim Zn = 0. 

n^oo 

Therefore, 

N{n,t) 

/(y/")) - /(yj")) = m(") + J2 ik{Lf{Yk-,) + Zk-i\ 

k=N{nfl)+l 

where Mj"^ = M^(„^() — M^(„ q). It is easy to verify that M^") is a martingale with respect 
to 

We also have 

t N(n,t) I N(n,t) \ 

/ L/(yW)(i.= lkLf{Yk.,)+[t- Y lAfiY^'^h 

■'^ k=n+l \ k=n+l j 
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Therefore 

Jo 

where P- hm i?!"-* = 0. It follows that any weak limit of the sequence (>'^"^)nGN solves the 
martingale problem associated with L. Prom this, together with the study of the stationary 
distribution of L (see Section . we will deduce Theorem |1] and Theorem |SJ <> 



3.3 The stationary distribution 

Theorem 6 The Markov process {Yt)t>o, on [0,+oo), with generator L has a unique 
stationary probability distribution u. Moreover, u has a density on [0, +oo), which vanishes 
on (0, r^] (where r^ = (1 —Pj^)/Pj^), and is positive and continuous on the open interval 
(r^j+oo). The stationary distribution v also satisfies the following property: for every 
compact set K in [0,+cxd), and every bounded continuous function f , we have 



lim sup 



^yifiYt))- j fdv 



0. 



(28) 



Before proving Theorem El we will show how Theorem 0] follows from (|28|) . 



Proof of Theorem m Pix t > 0. Por n large enough, we have < t < Yjlt=ilk-, so 
that there exists n G {!,... ,n — 1} such that 



5Z ik<t <^-ik- 

fc=n+l k=n 



Let tn = Efc=n+i Ik- We have 



< i — < 7n and Y^ 



in) 



Yr, 



Since t is fixed, the condition 7]^=^; 1 1 7fc < i implies lim n = oo and lim tn = t. 

^ n^oo n— ►oo 

Now, given e > 0, there is a compact set K such that for every weak limit /x of the 
sequence (yn)neN) /^(^^) < Using (PS)) . we choose t such that 



sup 

y&K 



< e. 



Now take a weakly convergent subsequence {Yn,,)keN- By another subsequence extraction, 
we can assume that the sequence (y converges weakly to a process y(°°) which satisfies 
the martingale problem associated with L. We then have, due to the quasi left continuity 

hm E/(y£7))=E/(y/~)), 

fc— >oo "fc 

for every bounded continuous function / (keep in mind that the functional tightness of 
(M^")) follows from Theorem 1.13 in [7] which in turn relies on the so-called Aldous 
criterion; any weak limiting process of such a sequence in the Skorokhod sense is then 
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quasi- left continuous and so is Y since B is pathwise continuous). Hence lim ]E/(1^^) = 

fc— »oo 

Kf{Y^ °°^). Observe that the law of 1^^°°^ is a weak limit of the sequence Yn, so that 
F{Y^°°^ G K") < £. Now we have 



so that, if n denotes the law of Yq 



(oo) 



lim sup 

fc^oo 



< 



fdiy 



Ey{fiYt))df,iy) 

< e + 2||/|UM^^) 

< e(l + 2| 



fdu 



It follows that any weak limit of the sequence (l^)ngN is equal to z^, which completes the 
proof of Theorem H) ^ 

For the proof of Theorem El we first observe that the generator L depends in an affine 
way on the state variable y. This affine structure suggests that the Laplace transform 
Ej^e-P^* has the form e-^f W+J/'/'pW, for some functions (pp and ipp. Affine models have been 
recently extensively studied in connection with interest rate modelling (see for instance ^ 
or [3]). The following proposition gives a precise description of the Laplace transform. 



Proposition 6 Let {Yt)t>o be the Markov process with generator L on [0, +oo). We have, 
forp>0,ye [0,+cx)), 

-P^* =exp((^p(t)+yVp(i)), 



E„e 



(29) 



where ipp is the unique solution, on [0, +oo) of the differential equation 

V'' = Pb — : Pa'^^ with V'(O) = -p, 
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and 



ipp{t) = {l-Pj^) / 'ippis)ds. 
Jo 

Before proving the Proposition, we study the involved ordinary differential equation. 
Lemma 8 Given ipo G (— oo,0], the ordinary differential equation 

V''=Pb^ PaV' (30) 

has a unique equation on [0,+oo) satisfying the initial condition 1^(0) = ipQ. Moreover, 
we have 

yt > 0, -0(0) < V(i)e''* < 0. 
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Proof: Existence and uniqueness of a local solution follows from the Cauchy-Lipschitz 
theorem. In order to prove non-explosion, observe that if ■0 solves (|8()|) . we have, using 
the inequality {e^'^ — l)/g > ip, 

+ TTlj)> 0. 

Therefore, the function t i-^ ip{t)e'^^ is non-decreasing, so that V'(O) ^ ^(t)e'^^. Since is 
an equilibrium of the equation, we have ^l'{t) < if ip{0) < 0, and the inequality is strict 
unless 0(0) = 0. Hence ip{0) < 'ip{t)e^^ < and the lemma follows easily. <> 

Proof of Proposition [HI Let Up{t,y) = exp(93p(t) +yipp{t)), where ipp and (pp are 
defined as in the statement of the Proposition. The existence of V'p follows from Lemma|Hl 
An easy computation shows that — Lup = on [0, +oo) x [0, +oo), so that, for T > 0, 
the process {up{T — t,Yt))Q^_^^rp is a martingale, and Kup{T,Yo) = E.Up{0,Yt), and the 
Proposition follows easily. ^ 

Proof of Theorem [HI 

• Uniqueness of the invariant distribution. We deduce from Lemma |H1 that, with the 

r+oo 

notation of PropositionEl |0p(i)l ^ ^iid lim (pp{t) = (l—p^) / ipp{s)ds. Therefore 

\imEyie-P^') = exp (^(1 - p^) ^p{s)ds^ , 

and the convergence is uniform on compact sets. This implies the uniqueness of the 
stationary distribution as well as (PH|). We also have the Laplace transform of v. 

j^^ e-Py^idy) = exp (^(1 - ^^(s)^^) . 

Note that, since ^pp < and ip'p = Pg^—^^ — P^^pi we have ■i/'p + Pa^^p ^ 0- Therefore, 
i'pit) < -pe~PA\ and 

Wp > 0, j e''Pyv{dy) > exp(-j>(l -Pa)/Pa) = exp{-pr^). 

This yields z^([0,r^)) = 0. 

6» Further properties of the invariant distribution u. The stationary distribution sat- 
isfies / Lfdu = for any continuously differentiable function / with compact support in 
[0,+oo). This reads 

V/ G Cj,, j (^^yZkMn/M + _ ^i^dy) = 0, (31) 

where r = p^/p^ and = (1 - p^)/pj^. 

We first show that z^({?'^}) = 0. Let 99 be a non-negative continuously differentiable 
function satisfying (/? = 1 in a neighbourhood of the origin and (/? = outside the interval 
[-1,1]. Forn > 1 let 

fn{y) = ^{n{y -rj), y G M. 
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We have fn{y) = if |y — > 1/n. In particular, the support of /„ hes in [0, +00), for 
n large enough. Applying (|3T|) with f = fn, '^e get 

Observe that lim fn = li^ \ so that 

n— >oo i A J 

J™o / y(f^(y+9) - fn{y))i^{dv) = (r, - 5')z^({':4 - 9}) - rAv{{rj,}) = -r^z^({r4}), 

where we have used z^(— cxD,r^) = 0. On the other hand, we have |(r^ — y)n (p' {n{y — r^))\ < 
sup„g]K(ti(/?'(n)), and lim {rnp'{n{y — r^))) = 0, so that, by dominated convergence, 

lim / {rA - y)n(p'{n{y - r^))v{dy) = 0. 



Hence v{{r^}) = 0. 

We now study the measure v on the open interval (r^, +00). Denote by V the set of 
all infinitely differentiable functions with compact support in (r^,+oo). We deduce from 
(jSU that, for / G P, 

^ u{dy)yf{y + 9) - - f v{dy)yf{y) + / y{dy){r^ - y)f'{y) = 0. (32) 



9-1 9 

Denote by Vg the measure defined by j Vg{dy)f{y) = j u{dy)f{y + g). We deduce 
from ()32() that satisfies the following equation in the sense of distributions: 

(y - -TaW + (1 - {r/g)y)v = --{y - g)ug, 

or 

/I - ir/g)y r y-g 
V H V = Vq. (33) 

y-^A gy-r^ 

Denote by F the function defined by 

Fiy) = e'y/3iy-rA)'''\ y > r^, (34) 

where d = rr^/g. We have 

F\y) = -'-^^Fiy), 

y-^A 



so that the equation satisfied by u reads 



F J F 

where the function G is defined by G{y) = — ^ 

y y A 
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On the set {r^,r^ + g), the measure Ug vanishes, so that u = XqF for some non 
negative constant Aq- At this point, we know that the restriction of the measure to the 
set (0, + g) has a density which vanishes on (0, r^) and is given by XqF on (r^, + g)- 

We wiU prove by induction that the distribution ly coincides with a continuous function 
on {r^,r^ + ng), which is infinitely differentiable on (r^ + (n — l)g,r^ + ng). The claim 
has been proved for n = 1. Assume that it is true for n. On the set (r^ , + (n + l)g), the 
distributional derivative of {1/F)iy coincides with the function y {G{y)/F{y))i'{y — g), 
which is locally integrable on + ng + g), continuous on (r^ + g,r^ + ng + g), and 

infinitely differentiable on (r^ +ng, +ng+g), due to the induction hypothesis (there may 
be a discontinuity at + (7 if d < 1). It follows that is a continuous (resp. infinitely 

differentiable) function, and so is on (r^ , + (n + l)^) (resp. (r^ + ng, r^+ng + g)). We 
have proved that v has a continuous density on (r^, +oo), which is infinitely differentiable 
on the open set U$^i('a + — 1)5; + "-5)- 

Finally, we prove that the density of is positive on (r^, +oo). Note that G{y) < if 
y > g and that the density vanishes at y — g ii y < g. Therefore (t'^^) < 0, so that the 
function y v{y)/F{y) is nondecr easing. It follows that Aq cannot be zero (otherwise 
u would be identically zero). Hence v{y) > for y £ {r^,r^ + g). Now, if i/(y) > for 
y £ {r^+ng — g, + ng), the function i//F \s strictly decreasing on (r^ + ng, r^+ng + g) 
and, therefore, cannot vanish. So, by induction, the density is positive on (r^, +oo). This 
completes the proof of Theorem [HI ^ 

Additional remarks. • The proof of Theorem |H1 provides a bit more information on the 
invariant distribution v. Let 5 > and let 0g denote its continuous density on (r^,+oo): 
the function is C°° on [r^, +00) \ (r^ + (7N) and it follows from ()34() and the definitions 
of r and (and d = rr^/g, see the proof of theorem E)) that 



(t>g{'rA) = +^ if 9>g*: </'s(?'a) ^ (0, +00) if g = g* and (j)g{r^) = Q if g < g* 



• One can characterize the finite positive exponential moments of v by slightly extending 
the proof of Proposition IS] (Laplace transform). For every y > 1, let 6{y) denote the unique 
(strictly) positive solution of the equation 



where g* = ^^^y^^ € (0, -^). As concerns the regularity of the density (f)g at points 
y E + gN, one easily derives from Equation ()33|) that for every m. A; € N, 

- (pg is C"""*"^ at + kg as soon as g < -i^^jp^, 

— the {m+ky^ derivative (pg^^'^^ is only right and left continuous at r^+kg if g = -^^ri- 




= y- 



Note that logy < 9{y) < 2{y — 1) and that lim 




1 and lim 




1. The 



result is as follows 




(36) 
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With the notations of Proposition |BJ it follows from Fatou's Lemma that 

Vp > 0, / ePyu{dy) < liminf Ey(eP^*). (37) 

We know that 

with (pp{t) = (1 — p^) /q '4>p{s)ds and is solution on the non-negative real line (if any) of 
^'(t) = G(^(t)), V(0) = P with G{u) = -p^u + ^(e^" - 1). 

The function G is convex on M+ and satisfies G(0) = G(p*) = 0, G((0,p*)) C (-cx),0). 
Let p£ (0,p*). The convexity of G implies 

r 1 G'(n) G(p) 
Vug 0,p, ^^<^i-^<0. 

G(p)t 

It follows that "(/^p does exist on and satisfies < ippit) < pe p (hence it goes to 
when t goes to infinity). One derives that 

lim ^p{t) = (1 -pj / Mt)dt < -(1 - Pa): 



t"Too^'^'^ ,,4,^^ .p,.,^. _ -^'G(p)- 

Combining this with (|37() yields 

/2 
e^'V((iy) < e"^^"^^^^ < +oo. 

On the other hand if p = p*, Tpp{t) = p* and (pp{t) = (1 — Pj^)p*gt. Consequently 

Vt > 0, / eP>v{dy) = I ¥.y{eP9^')v{dy) = e^^-PA^* [ eP>u{dy). 



Now the right hand side of this equality goes to oo as t goes to infinity since (1 — PA)Pg >0 
which shows that j e^9^z^(dy) = +oo (since it cannot be 0). 

• One has, in accordance with the convergence rate result obtained for p„ = o(7„), that 



yv{dy)-- ^ 



vr 



To prove this claim, one first notes, using the definition H18() of the generator L, that 

L(Id){y) = l — p^—iry. Hence the above claim will follow from J L[Id)[y)v{dy) = 0. Let 

if : R+ — > M_|_ denote a continuously differentiable function such that (/?(y) = y if y G [0; 1]; 
</?(y) = if y > 2 and ip' is bounded on M_|_. Set ^Pn{y) = nipiu/n), re > 1. One checks 



29 



Figure 1: Graphs of the p.d.f (pg, = 2/5, g = 1; the vertical dotted line shows the mean 
^ of V. Left: p^ = 1/3 (g* > g = I). Center: p^ = 4/15 (g* = g = I). Right: 
p^ = 1/6 (g* <g = I). 



that L{ipn) — > L{Id) as n goes to infinity and \L{ipn){y)\ < ay + h for some positive real 
constants a, h. One derives by the dominated convergence theorem that 

L{Id){y)u{dy) = lim / L{^r.){y)i^{dy) = 



where we used that the function ipn has compact support on [0, +cxd). One shows similarly 
that j Liu ^ u'^){y)u{dy) = to derive that 



[y-^—^^\{dy)=g 



2^2 



Note that, as one could expect, this variance goes to as (7 —> 0. As a conclusion, 
we present in figure 1 three examples of shape for (pg. They were obtained from an exact 
simulation of the Markov process (lt)j>o (associated to the generator L) at its jump times: 
we approximated the p.d.f. by a histogram method using Birkhoff 's ergodic Theorem. 

Figures should be here 

A final remark about the case tt = and 'jn = 9 Pn- In that setting (see Remark^ 
the asymptotics of the algorithm cannot be elucidated by using the ODE approach since 
it holds in a weak sense. Setting y„ = 1 — 2X„ one checks that y„ G [—1)1] and 

Yn+i = y„(l - 2gp2^i(l -p^)) - 2gpn+iAMn+i 

and that K{{AMn+i)'^ \J^n+i) = — ^n) + ^(Pn+i)- Then, a similar approach as 

that developed in this section (but significantly less technical since (Yn) is bounded by 1) 
shows that Yn converges in distribution to the invariant distribution fi of the Brownian 
diffusion with generator Cf{y) = —2g{\ — PA)y f {v) + ^d'^PAi^ ~ y'^)f"{y)- tha,t case, it 
is well-known that /i has a density function for which a closed form is available (see jS]), 
namely 

fi{dy) = m{y)dy with m{y) = Cg,r^(l - y ) 9 ' l(_i_i)(y). 
Note that when g = 2r^ = 2{l/p^ — 1) > 0, /u is but the uniform distribution over [—1, 1]. 
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